Apparatus and method for obtaining object-color component data

ABSTRACT

First image data is obtained by performing an image capturing operation with a flash and, subsequently, second image data is obtained by performing an image capturing operation without a flash. By using data of a differential image between the first and second image data and a relative spectral distribution of flash light, the spectrum reflectivity in a position on a subject corresponding to each pixel is obtained and object-color component data is acquired as data from which an influence of an illumination environment has been removed. On the other hand, illuminant component data indicative of spectral distributions of a plurality of illumination light are prepared. By combining arbitrary illuminant component data to the obtained object-color component data, an image with a different illumination environment can be reproduced.

This application is based on application No. 11-247010 filed in Japan,the contents of which are hereby incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to a technique for obtaining image data ofa subject and, more particularly, to a technique of obtaining image databy a digital camera or the like.

2. Description of the Background Art

Hitherto, image processes are performed to correct the hue or atmosphereof an image obtained as digital data by an image input device such as adigital camera. A representative one of such processes is a colorcorrection based on white balance. In the correction based on whitebalance, an image is corrected based on the overall balance of the colorof the image so that a white body will appear to be white. This allowsthe influence of the color of illuminant light on the subject to beremoved from the image to some extent, and the image is corrected to onethat agrees with the eyesight of human beings.

The conventional correction of the hue of an image is performeduniformly on the entire image. The data of the original image and datarelated to the correction of the hue are not therefore separatelytreated. For example, the corrected image is integrally stored asbrightness information of RGB.

On the other hand, in some cases, it is desired to correct the hue of animage in order to impart a sense given by the image to an observer toanother image. To be specific, in some cases, it is desired to use theatmosphere produced by the illumination environment at the time ofcapturing an image in an image captured in another illuminationenvironment. The illumination environment is defined here as anenvironment related to illumination in which not only thecharacteristics of a light source but also the conditions around thesubject are taken into consideration.

Since the data of an image is, however, conventionally treated asintegral data, the atmosphere produced by the illumination environmentin an image cannot be used by another image. Further, in the case oftrying to achieve an environment produced by a specific illuminationenvironment by correcting the hue of the image, an unnatural image isproduced only by uniformly changing the hue of the entire image.

SUMMARY OF THE INVENTION

The present invention is directed to a digital image capturingapparatus.

According to an aspect of the present invention, an image capturingapparatus comprises: an illumination unit for changing an illuminationenvironment around a subject; an image capturing part for obtaining animage of the subject; a first memorizing part for memorizing first imagedata obtained by the image capturing part before illuminating by theillumination unit; a second memorizing part for memorizing second imagedata obtained by the image capturing part with illuminating by theillumination unit; and a subject data generating part for generatingsubject data on the basis of the first image data, the second imagedata, and changing degree of the illumination environment by theillumination unit, the subject data corresponding to image data fromwhich influence of the illumination environment is removed.

In another aspect of the present invention, an image capturing apparatuscomprises: an image capturing part for obtaining an image of a subject;at least one filter being capable of existing on an optical path of theimage capturing part; a memorizing part for memorizing a plurality ofimage data obtained by the image capturing part with changingarrangement of the at least one filter; and a subject data generatingpart for generating subject data on the basis of the plurality of imagedata memorized in the memorizing part and spectral transmittance of theat least one filter, the subject data corresponding to image data fromwhich influence of an illumination environment is removed.

Since the subject data corresponds to image data from which theinfluence of an illumination environment is removed, by combining theillumination data corresponding to the influence of the illuminationenvironment exerted on the image and the subject data, data of an imageunder a desired illumination environment can be generated.

The present invention is also directed to a digital image capturingmethod, a digital image processing device, and a computer-readablemedium carrying a program for processing image data.

These and other objects, features, aspects, and advantages of thepresent invention will become more apparent from the following detaileddescription of the present invention when taken in conjunction with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view illustrating a whole digital cameraaccording to a first embodiment;

FIG. 2 is a rear view of the digital camera shown in FIG. 1;

FIG. 3 is a block diagram showing the construction for executing animage process in the digital camera illustrated in FIG. 1;

FIG. 4 is a block diagram showing the functions of the construction ofFIG. 3;

FIG. 5 is a flow chart showing a flow of operations for obtaining imagedata in the digital camera illustrated in FIG. 1;

FIG. 6 is a flow chart showing a flow of operations of a light emissioncontrol circuit;

FIGS. 7 and 8 are schematic diagrams showing the relation betweenilluminant component data and sense information;

FIG. 9 is a flow chart showing a flow of operations for reproducing animage in the digital camera illustrated in FIG. 1;

FIGS. 10 and 11 are views each showing an example of a screen on which alist of sense information is displayed;

FIGS. 12 to 14 are views each showing an example of a spectraldistribution of illuminant light represented by the illuminant componentdata;

FIG. 15 is a perspective view of a whole digital camera according to asecond embodiment;

FIG. 16 is a block diagram showing the construction for executing animage process in the digital camera of FIG. 15;

FIG. 17 is a block diagram showing the functions of the construction ofFIG. 16;

FIG. 18 is a flow chart showing a flow of operations for obtaining imagedata in the digital camera illustrated in FIG. 15;

FIG. 19 is a diagram showing spectral transmittance of an on-chip filteron a CCD and that of a movable filter;

FIG. 20 is a diagram showing spectral transmittance in the case wherelight transmits both the on-chip filter on the CCD and the movablefilter;

FIG. 21 is a perspective view showing a whole digital camera accordingto a third embodiment;

FIG. 22 is a block diagram showing the functional construction of thedigital camera of FIG. 21;

FIG. 23 is a flow chart showing a flow of operations for obtaining imagedata in the digital camera of FIG. 21;

FIG. 24 is a flow chart showing a flow of the whole operations of adigital camera according to a fourth embodiment;

FIG. 25 is a block diagram showing the construction in the case ofautomatically switching an image capturing mode;

FIG. 26 is a block diagram showing the construction for obtaining arelative spectral distribution of flash light by interpolating a groupof flash spectral data;

FIG. 27 is a diagram showing an example of obtaining a relative spectraldistribution of flash light by an interpolation;

FIG. 28 is a flow chart showing a flow of operations for obtaining imagedata in a fifth embodiment;

FIG. 29 is a block diagram showing the construction of determining arelative spectral distribution of flash light by referring to adatabase;

FIG. 30 is a flow chart showing a flow of operations for obtaining imagedata in a sixth embodiment;

FIG. 31 is a diagram showing the construction of an image data obtainingsystem in a seventh embodiment; and

FIG. 32 is a diagram showing a modification of the digital camera ofFIG. 15.

DESCRIPTION OF THE PREFERRED EMBODIMENTS 1. First Embodiment

FIG. 1 is a perspective view showing a whole digital camera 1 as adigital image capturing apparatus according to a first embodiment. Thedigital camera 1 comprises a lens unit 11 for capturing an image and amain body part 12 for processing an image obtained as digital data bythe lens unit 11.

The lens unit 11 has a lens system 111 having a plurality of lenses anda CCD 112 for capturing an image of a subject via the lens system 111.An image signal outputted from the CCD 112 is sent to the main body part12. In the lens unit 11, a finder 113 used by the operator to capturethe subject, a range sensor 114, and the like are also arranged.

In the main body part 12, a flash 121 and a shutter button 122 areprovided. When the operator captures the subject via the finder 113 andoperates the shutter button 122, an image is obtained electrically bythe CCD 112. At this time, the flash 121 is used in accordance withnecessity. The CCD 112 is 3-band image capturing means for obtainingvalues related to each of the colors of R, G and B as values of eachpixel.

An image signal from the CCD 112 is subjected to processes which will bedescribed hereinlater in the main body part 12 and a resultant signal isstored into an external memory 123 (what is called a memory card)attached to the main body part 12 as necessary. The external memory 123is ejected from the main body part 12 by opening a lid on the under faceof the main body part 12 and operating an ejection button 124. Datastored in the external memory 123 as a recording medium can betransferred to a device such as a computer separately provided. On thecontrary, the digital camera 1 can read the data stored in the externalmemory 123 by another device.

FIG. 2 is a rear view of the digital camera 1. In the center of the rearface of the main body part 12, a liquid crystal display 125 fordisplaying a captured image or a menu for the operator is provided. Anoperation button 126 for performing an input operation in accordancewith the menu displayed on the display 125 is disposed on a side of thedisplay 125. By the operation button 126, operation of the digitalcamera 1, setting of image capturing conditions, control of the externalmemory 123, reproduction of an image which will be describedhereinafter, and the like can be performed.

FIG. 3 is a block diagram schematically showing a construction formainly executing an image process in the construction of the digitalcamera 1.

In the construction shown in FIG. 3, the lens system 111, the CCD 112,an A/D converting part 115, the shutter button 122, a CPU 21, a ROM 22and a RAM 23 realize a function of obtaining an image. Specifically,when an image of a subject is formed on the CCD 112 by the lens system111 and the shutter button 122 is pressed, an image signal from the CCD112 is converted into a digital signal by the A/D converting part 115.The digital image signal obtained by the A/D converting part 115 isstored as image data into the RAM 23 of the main body part 12. Thecontrol of these processes is carried out by the CPU 21 operating inaccordance with a program 221 stored in the ROM 22.

The CPU 21, the ROM 22 and the RAM 23 disposed in the main body part 12realize a function of processing an image. Specifically, the CPU 21performs an image process on the obtained image in accordance with theprogram 221 stored in the ROM 22 while utilizing the RAM 23 as a workarea.

The external memory 123 is connected to the RAM 23 and various data istransferred on the basis of an input operation by the operation button126. The display 125 switches and displays an image and information tothe operator in response to a signal from the CPU 21.

The flash 121 is connected to the CPU 21 via a light emission controlcircuit 121 a. When an instruction of turning on the flash 121 isreceived from the CPU 21, the light emission control circuit 121 aperforms a control to suppress variations in the light emittingcharacteristic of the flash 121 in image capturing operations, so that aspectral distribution (spectral intensity) of light from the flash 121is controlled to be uniform.

FIG. 4 is a block diagram showing a construction of functions realizedmainly by the CPU 21, the ROM 22 and the RAM 23 together with the otherconstruction. FIG. 5 is a flowchart showing a flow of image capturingand image processes. In the construction shown in FIG. 4, a differentialimage generating part 201, an object-color component data generatingpart 202, an illuminant component data generating part 203 and an imagereproducing part 204 are the functions realized by the CPU 21, the ROM22, the RAM 23 and the like. The operation of the digital camera 1 willbe described hereinbelow with reference to the drawings.

First, image capturing is performed with the flash and an image of thesubject irradiated with the flash light is obtained (hereinbelow, calleda “first image”). To be specific, an image is obtained with the flash121 by the CCD 112, the obtained image (accurately, an image signal) issent from the A/D converting part 115 to the RAM 23 and is stored asfirst image data 231 (step ST11).

Then, image capturing is performed without the flash and an image of thesubject under an illumination environment without using the flash light(hereinbelow, called a “second image”) is obtained. In other words, animage is obtained by the CCD 112 without using the flash, and theobtained image is sent from the A/D converting part 115 to the RAM 23and stored as second image data 232 (step ST12).

The image capturing operations of twice are performed quicklysuccessively. The first and second images are therefore captured in thesame image capturing range. The image capturing operations of twice areperformed under the same conditions of the shutter speed (integrationtime of the CCD 112) and the aperture.

The light emission of the flash 121 is controlled by the light emissioncontrol circuit 121 a so that the spectral distribution of flash lightbecomes uniform. FIG. 6 is a flow chart showing the flow of operationsof the light emission control circuit 121 a.

When image capturing is performed with the flash, or prior to imagecapturing, first, the light emission control circuit 121 a startsmonitoring a charging voltage to the power source of the flash 121 (thatis, a voltage applied to the flash 121) (step ST21). When it isrecognized that the charging voltage reaches a predetermined voltage(for example, 330V) (step ST22), a power is supplied from the powersource to the flash 121 and light emission is started (step ST23).

Upon start of the light emission, the light emission control circuit 121a starts monitoring light emission time (step ST24). After that, when itis confirmed that predetermined time has elapsed since the start oflight emission (step ST25), the light emission is stopped (step ST26).

As described above, the light emission of the flash 121 is controlled bya constant voltage and light emission time, so that the light emissioncharacteristic does not vary in image capturing operations. The spectraldistribution of the flash 121 is kept to be uniform by the lightemission control, preliminarily measured and stored as flash spectraldata 234 in the RAM 23. To be accurate, a relative spectral distributionof flash light (spectral distribution normalized by setting the maximumspectral intensity to 1 and will be called hereinbelow a “relativespectral distribution”) is used as the flash spectral data 234.

After the first image data 231 and the second image data 232 is storedinto the RAM 23 by the image capturing operations of twice, thedifferential image generating part 201 subtracts the second image data232 from the first image data 231 to thereby obtain differential imagedata 233. By the operation, the values of R, G and B of each pixel inthe second image are subtracted from the values of R, G and B of acorresponding pixel in the first image, so that a differential imagebetween the first and second images is obtained (step S13).

Then, components obtained by removing the influence of the illuminationenvironment from the second image by using the differential image data233 and the flash spectral data 234 are obtained as object-colorcomponent data 235 by the object-color component data generating part202 and stored into the RAM 23 (step ST14). The object-color componentdata 235 is data substantially corresponding to the spectralreflectivity of the subject. The principal of obtaining the spectralreflectivity of the subject will be explained hereinbelow.

First, a spectral distribution of illumination light (illumination lightunder illumination environment including both direct light from a lightsource and indirect light) for illuminating the subject is set as E(λ)and the spectral distribution E(λ) is expressed as follows by usingthree basis functions E₁(λ), E₂(λ) and E₃(λ) and weighting coefficientsε₁, ε₂, and ε₃. $\begin{matrix}{{E(\lambda)} = {\sum\limits_{i = 1}^{3}{ɛ_{i}{E_{i}(\lambda)}}}} & (1)\end{matrix}$

Similarly, the spectral reflectivity S(λ) in a position on the subjectcorresponding to a pixel (hereinbelow, called a “target pixel”) isexpressed as follows by using three basis functions S₁(λ), S₂(λ) andS₃(λ) and weighting coefficients σ₁, σ₂ and σ₃. $\begin{matrix}{{S(\lambda)} = {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}}}} & (2)\end{matrix}$

Light I(λ) incident on the target pixel on the CCD 112 (incident lightin the case of ignoring a filter and the like in the lens unit 11) isexpressed as follows. $\begin{matrix}{{I(\lambda)} = {\sum\limits_{i = 1}^{3}{ɛ_{i}{{E_{i}(\lambda)} \cdot {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}}}}}}} & (3)\end{matrix}$

When a value related to one of the colors R, G and B of the target pixel(hereinbelow, called a “target color”) is set as ρ_(c) and the spectralsensitivity of the target color on the CCD 112 is R_(c)(λ), the valueρ_(c) is derived from the following equation.ρ_(c) =∫R _(c)(λ)I(λ)dλ  (4)

In the case where the value of the target color of the target pixel inthe first image captured with the flash is ρ_(c1) and the correspondingvalue in the second image captured without the flash is ρ_(c2), thecorresponding value ρ_(s) in the differential image is obtained by thefollowing equation. $\begin{matrix}\begin{matrix}{\rho_{s} = {\rho_{c1} - \rho_{c2}}} \\{= {~~}{\int{{R_{c}(\lambda)}\left\{ {{I_{1}(\lambda)} - {I_{2}(\lambda)}} \right\}{\mathbb{d}\lambda}}}} \\{= {\int{{R_{c}(\lambda)}\left\{ {\sum\limits_{i = 1}^{3}{\left( {ɛ_{1i} - ɛ_{2i}} \right){{E_{i}(\lambda)} \cdot {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}}}}}} \right\}{\mathbb{d}\lambda}}}} \\{= {\sum\limits_{i = 1}^{3}{\sum\limits_{j = 1}^{3}{ɛ_{si}\sigma_{j}\left\{ {\int{{R_{c}(\lambda)}{E_{i}(\lambda)}{S_{j}(\lambda)}{\mathbb{d}\lambda}}} \right\}}}}}\end{matrix} & (5)\end{matrix}$where, I₁(λ) denotes light incident on the target pixel in the casewhere the flash is used, ε₁₁, ε₁₂ and ε₁₃ are weighting coefficients ofbasis functions related to illumination light including flash light,similarly, I₂(λ) denotes light incident on the target pixel in the casewhere no flash is used, and ε₂₁, ε₂₂ and ε₂₃ are weighting coefficientsof basis functions related to illumination light which does not includeflash light. ε_(si)(i=1, 2, 3) is equal to (ε_(1i)−ε_(2i)).

In the equation 5, the basis functions E_(i)(λ) and S_(j)(λ) arepredetermined functions and a spectral sensitivity R_(c)(λ) is afunction which can be preliminarily obtained by measurement. Theinformation is prestored in the ROM 22 and the RAM 23. On the otherhand, since the shutter speed (or integration time of the CCD 112) andthe aperture are controlled to be the same in the image capturingoperations of twice, the differential image obtained by subtracting thesecond image from the first image corresponds to an image influenced byonly the illumination environment, that is, an image irradiated withonly the flash light as illumination light. Consequently, the weightingcoefficient ε_(si) can be derived from the relative spectraldistribution of flash light by a method which will be describedhereinlater.

In the equation 5, therefore, only the three weighting coefficients σ₁,σ₂ and σ₃ are unknown. The equation 5 can be obtained with respect toeach of the three colors R, G and B in a target pixel. By solving thethree equations, the three weighting coefficients σ₁, σ₂ and σ₃ can beobtained. That is, the spectrum reflectivity in a position on thesubject corresponding to the target pixel can be obtained.

The method of obtaining the weighting coefficient ε_(si) will now bedescribed. As described above, the differential image corresponds to animage irradiated with only flash light as illumination light and therelative spectral distribution of illumination light in the differentialimage is known. On the other hand, the subject in an area far from theflash 121 is irradiated with less flash light than that in an area nearthe flash 121. The further the area from the flash 121 is, therefore,the darker the differential image becomes.

While maintaining the relative relation of the three weightingcoefficients ε_(s1), ε_(s2) and ε_(s3), the values of the weightingcoefficients increase or decrease in proportional to the brightness ofthe target pixel in the differential image. That is, when the brightnessof the target pixel in the differential image is low, the weightingcoefficients ε_(s1), ε_(s2) and ε_(s3) are determined as small values.When the brightness is high, the weighting coefficients ε_(s1), ε_(s2)and ε_(s3) are determined as large values. The relative relation of thethree weighting coefficients ε_(s1), ε_(s2) and ε_(s3) is preliminarilyobtained so that the total weight of the three basis functions E₁(λ),E₂(λ) and E₃(λ) is proportional to the spectral distribution of flashlight. The proportional relation between the brightness and theweighting coefficient ε_(si) is preliminarily obtained by measurement.

The weighting coefficient ε_(si) is a value indicative of the spectraldistribution of coefficients ε₂₁, ε₂₂, and ε₂₃ are calculated and theobtained three weighting coefficients are used as the illuminationcomponent data 236 (step ST15). Thus, the illumination component data236 becomes values which does not depend on the position of a pixel. Bycombining the illumination component data 236 with the otherobject-color component data 235 as will be described hereinlater, theatmosphere of the illumination environment can be brought into an imageof another subject.

Before the illumination component data 236 is stored, informationindicative of the sense (hereinbelow, called “sense information”) isadded to the illumination component data 236 on the basis of an input ofthe operator via the operation button 126 (step ST16). Specifically,sense information 236 a is imparted to the illumination component data236 as shown in FIG. 7 or related to the illumination component data 236as shown in FIG. 8. As described above, the illumination component data236 is data indicative of an influence of the illumination environmentin an image. An image reproduced by an operation which will be describedhereinlater by using the illumination component data 236 gives a certainkind of impression to the observer. In the digital camera 1, theilluminant component data 236 is stored in a recognizable state by usinginformation indicating the impression (feeling) of the observer.

Specifically, information such as a word indicating the sense of aseason such as “spring-like” or “midsummer”, a word indicating the senseof time such as “evening glow”, “early morning” or “early afternoon”, aword indicating the sense of temperature such as “chilly” or “glowinghot”, or a word indicating the sense of weather such as “foggy” or“cloudy” is imparted to the illuminant component data 236 and theresultant is stored.

When the object-color component data 235 and the illuminant componentdata 236 are obtained, the data is transferred and stored into theexternal memory 123 which is the flash light emitted to a position onthe subject corresponding to the target pixel and also indicative of thespectral distribution of an amount of a change of the illumination lightby the flash 121 between the first and second images. A process ofobtaining the weighting coefficient ε_(si) from the flash spectral data234, therefore, corresponds to a process of obtaining a degree of achange in spectrum of the illumination environment (illumination light)with the flash 121 from the relative spectral distribution of the flashlight.

On the basis of the principle, the object-color component datagenerating part 202 in the digital camera 1 obtains the spectrumreflectivity in a position on the subject corresponding to each pixelwith reference to the pixel value of the differential image data 233 andthe flash spectral data 234. The spectral reflectivity of the subjectcorresponds to image data from which the influence of the illuminationenvironment is removed and is stored as the object-color component data235 into the RAM 23 (step ST14).

When the object-color component data 235 is obtained, three equationsrelated to weighting coefficients ε₂₁, ε₂₂ and ε₂₃ can be obtained onthe basis of the values of R, G and B of each pixel in the second imageby the equations 3 and 4. In the illumination component data generatingpart 203, the weighting coefficient ε_(2i) related to each pixel in thesecond image is obtained by solving the equations. The obtainedweighting coefficient ε_(2i) of each pixel is a component indicative ofthe influence of the illumination environment without the flash light inthe second image.

The weighting coefficient ε_(2i) of each pixel may be used asillumination component data 236. In the case of an illuminationenvironment of almost even illumination light, the weightingcoefficients ε_(2i) of pixels do not vary so much. Consequently, averagevalues of all of pixels with respect to each of the weighting detachablefrom the main body of the digital camera 1 (step ST17). The object-colorcomponent data 235 and/or the illumination component data 236 may bestored in the external memory 123 in a state where each data can be readindependently.

The operation of the digital camera 1 when an image is reproduced byusing the object-color component data 235 and the illumination componentdata 236 stored in the external memory 123 as described above will beexplained by referring to FIG. 9. It is assumed that a plurality ofobject-color component data 235 and a plurality of illuminant componentdata 236 are prestored in the external memory 123. The plurality ofobject-color component data 235 and the plurality of illuminantcomponent data 236 may be stored in the external memory 123 from anexternal computer.

First, the operator gives an instruction via the operation button 126while watching the contents on the display 125 to thereby read desiredobject-color component data 235 from the external memory 123 and writeit to the RAM 23 (step ST31). By the operation, an object of an image tobe reproduced (that is, the subject at the time of image capturing) isdetermined.

The sense information 236 a imparted to each of the illuminant componentdata 236 stored in the external memory 123 is read out and a list of aplurality of pieces of sense information 236 a is displayed on thedisplay 125 (step ST32). The illuminant component data 236 as candidatesto be selected in the external memory 126 becomes selectable on thebasis of the sense information 236 a. FIGS. 10 and 11 are diagrams ofexamples of display screens of sense information of the display 125 inthe case where light sources are shown and in the case where lightsources are not shown, respectively.

FIGS. 12 to 14 are diagrams each showing an example of a spectraldistribution of illumination light common to each pixels expressed bythree weighting coefficients of the illumination component data 236 andpredetermined three basis functions in terms of relative spectrumintensity. As the sense information, a word such as “evening glow” isgiven to FIG. 12, a word such as “chilly” is given to FIG. 13, and aword such as “early afternoon” is given to FIG. 14. Obviously, it isunnecessary to give information indicating the sense to all theilluminant component data 236 but a word such as “fluorescent lamp” maybe imparted. The sense information is not limited to words but may be acolor, a sound or the like.

The operator designates sense information from the list of the senseinformation 236 a by using the operation button 126. On receipt of thedesignation of the operator, the image reproducing part 204 determinesthe illuminant component data 236 to which the designated senseinformation 236 a is imparted as illuminant component data to be usedfor reproduction (step ST33), reads the illuminant component data 236from the external memory 123 and stores it into the RAM 23 (step ST34).

In the case of reproducing an image as the base of the object-colorcomponent data 235 which has been already stored, the illuminantcomponent data 236 at the time when the object-color component data 235is obtained is stored into the RAM 23. In the case of reproducing theimage as an image under the illumination environment of another image,the illuminant component data 236 obtained when the another image iscaptured is stored into the RAM 23.

After that, the object-color component data 235 and the illuminantcomponent data 236 are combined by the image reproducing part 204 byarithmetic operations shown in the equations 3 and 4, thereby obtainingvalues ρ_(r), ρ_(g) and ρ_(b) of R, G and B of each pixel, and imagedata as an object is generated (step ST35). The obtained image data isdisplayed on the display 125 provided on the rear face of the digitalcamera 1 (step ST36).

The reproduced image is stored in the external memory 123 in a regularimage format as necessary (which may be compressed).

As described above, the digital camera 1 can obtain the object-colorcomponent data 235 corresponding to image data from which the influenceof the illumination environment is removed from the first image capturedwith the flash, the second image captured without the flash, and therelative spectral distribution of flash light. That is, the object-colorcomponent data 235 indicative of the spectrum reflectivity of the objectcan be easily derived from two images obtained under differentillumination environments. The illumination environment can be easilychanged by using the flash 121.

Also, the digital camera 1 capable of obtaining the object-colorcomponent data 235 by a simple change in the specification of thedigital camera having a CCD provided with a general on-chip filter canbe realized without having a special mechanism. Consequently, theoperation of obtaining the object-color component data 235 can berealized as a special mode of the general digital camera 1 withoutincreasing the manufacturing cost.

In the digital camera 1, the illuminant component data 236 is obtainedfrom the derived object-color component data 235 and the second imagedata. In the digital camera 1, therefore, by properly combining theselected one of the plurality of illuminant component data 236 to theobject-color component data 235, a desired image under a desiredillumination environment can be reproduced. That is, in the case ofreproducing an image in an atmosphere at the time when the image iscaptured, it is sufficient to combine the object-color component data235 and the illuminant component data 236 generated as a pair. In thecase of reproducing an image by using the illumination environment ofanother image, the illuminant component data 236 obtained under anotherillumination environment is combined to the object-color component data235, thereby reproducing an image under another illuminationenvironment.

The image data as a base of obtaining the illumination component data236 is not limited to the second image data but may be the first imagedata. Further, image data captured by using the same object separatelymay be used.

By providing standard light (D65 or the like) as the illuminantcomponent data 236 as exemplified in FIG. 10, an image obtained under anarbitrary illumination environment is reproduced by using data of thestandard light and accurate color reproduction of the subject can bemade from the obtained image. Consequently, an image suitable forprinting, Internet shopping and the like can be generated.

Further, in the digital camera 1, the sense information 236 a isimparted to the illumination component data 236 and the operator canselect the illumination component data 236 on the basis of the senseinformation 236 a, so that reproduction of an image based on sense canbe enjoyed.

It has been described above that the weighting coefficients σ₁, σ₂ andσ₃ corresponding to pixels are stored as the object-color component data235 and the weighting coefficients ε₁, ε₂ and ε₃ common to the pixels(peculiar to pixels in the case where general versatility is notdemanded) are stored as the illuminant component data 236. They may bestored together with the basis functions S₁(λ), S₂(λ) and S₃(λ) of thespectrum reflectivity of the subject and the basis functions E₁(λ),E₂(λ) and E₃(λ) of the spectral distribution of illumination light.

2. Second Embodiment

FIG. 15 is a diagram showing the construction of a digital camera 1 aaccording to a second embodiment. As shown in FIG. 15, the digitalcamera 1 a has a filter 116 in front of the CCD 112 of the digitalcamera 1 in the first embodiment and the filter 116 is movably disposedby the operation of a motor 117. The other construction is similar tothat of the first embodiment and the same components as those of FIG. 1are designated by the same reference numerals.

In the digital camera 1 a, the filter 116 swings around the rotary shaftof the motor 117 as a center between a position on the optical axis ofthe lens system 111 (that is, on an optical path at the time of imagecapturing) and a position deviated from the optical path for imagecapturing. In the second embodiment, object-color component datacorresponding to image data from which the influence of illuminance isremoved and illuminant component data indicative of the influence of theillumination environment is obtained by using an image in the case wherethe filter 116 is positioned on the optical path for image capturing andan image in the case where the filter 116 is not positioned on theoptical path for image capturing.

The CCD 112 is 3-band input image capturing means of acquiring values ofR, G and B in a manner similar to the first embodiment. The digitalcamera 1 a has an ordinary image capturing mode and a special imagecapturing mode. In the ordinary image capturing mode, image dataconsisting of values of R, G and B is obtained (that is, an operationsimilar to that of a commercially available digital camera isperformed). In the special image capturing mode, the object-colorcomponent data and the illuminant component data are obtained.

FIG. 16 is a block diagram schematically showing the construction mainlyfor executing an image process in the construction of the digital camera1 a. Except for the point such that the filter 116 and the motor 117 areprovided as shown in FIG. 16, the internal construction of the digitalcamera 1 a is similar to that of the first embodiment. Processesperformed by the CPU 21, the ROM 22, the RAM 23 and the like aredifferent from those of the first embodiment and will be described indetail hereinlater. The flash 121 is used to perform the ordinary imagecapturing and is not used to perform the image capturing for obtainingthe object-color component data and the like. The light emission controlcircuit 121 a does not perform a light emitting control make the lightemission characteristic uniform as in the first embodiment.

FIG. 17 is a block diagram showing a component data generating part 205for obtaining the object-color component data and the illuminantcomponent data in the digital camera 1 a in the special image capturingmode together with the peripheral construction. The component datagenerating part 205 is realized by the CPU 21, the ROM 22, the RAM 23and the like. Specifically, the component data generating part 205 is afunction realized by the operation of the CPU 21 while using the RAM 23as a work area in accordance with the program 221 in the ROM 22. FIG. 18is a flow chart showing the flow of operations of the digital camera 1 ain the special image capturing mode.

In the special image capturing mode, first, an image is captured bysetting the filter 116 in a position deviated from the image capturingoptical path. Consequently, a first preliminary image which is a colorimage is stored as the first image data 231 into the RAM 23 via the A/Dconverting part 115 (step ST41).

The filter 116 is moved onto the image capturing optical path by theaction of the motor 117 and an image is captured via the filter 116.Consequently, a second preliminary image which is a color image (ofcolors which are not actual colors of the subject since the filter 116is used) is stored as the second image data 232 into the RAM 23 (stepST42). The operations in steps ST41 and ST 42 are promptly performedconsecutively to capture images of the same subject.

After the first and second image data 231 and 232 are obtained, theobject-color component data 235 corresponding to image data from whichthe influence of the illumination environment has been removed and theilluminant component data 236 indicative of the influence of theillumination environment are obtained by the component data generatingpart 205, and these data are stored into the RAM 23 (step ST43). Thedetails of the processes of the component data generating part 205 willbe described hereinlater.

After that, in a manner similar to the first embodiment, senseinformation is imparted to the illuminant component data 236 (step ST44)and the object-color component data 235 and the illuminant componentdata 236 are stored into the external memory 123 (step ST45).

The principle of the operation of the component data generating part 205in step ST43 will now be explained.

As described above, in the case of expressing the spectral distributionE(λ) of illumination light and the spectrum reflectivity S(λ) in aposition on the subject corresponding to a certain pixel (target pixel)by using three basis functions and weighting coefficients as shown inthe equations (1) and (2), light I(λ) entering the digital camera 1 afrom the position on the subject corresponding to the target pixel isexpressed by the equation (3). When the spectral sensitivitiescorresponding to the R, G and B colors on the CCD 112 are set asR_(r)(λ), R_(g)(λ) and R_(b)(λ), respectively, the R, G and B valuesρ_(r), ρ_(g) and ρ_(b) of a target pixel are obtained by the equation(6) in a manner similar to the equation (4). $\begin{matrix}\begin{matrix}{\rho_{r} = {~~}{\int{{R_{r}(\lambda)}{\sum\limits_{i = 1}^{3}{ɛ_{i}{{E_{i}(\lambda)} \cdot {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}{\mathbb{d}\lambda}}}}}}}}} \\{= {\sum\limits_{i = 1}^{3}{\sum\limits_{j = 1}^{3}{ɛ_{i}\sigma_{j}\left\{ {\int{{R_{r}(\lambda)}{E_{i}(\lambda)}{S_{j}(\lambda)}{\mathbb{d}\lambda}}} \right\}}}}} \\{\rho_{g} = {\int{{R_{g}(\lambda)}{\sum\limits_{i = 1}^{3}{ɛ_{i}{{E_{i}(\lambda)} \cdot {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}{\mathbb{d}\lambda}}}}}}}}} \\{= {\underset{i = 1}{\overset{3}{\;\sum}}{\sum\limits_{j = 1}^{3}{ɛ_{i}\sigma_{j}\left\{ {\int{{R_{g}(\lambda)}{E_{i}(\lambda)}{S_{j}(\lambda)}{\mathbb{d}\lambda}}} \right\}}}}} \\{\rho_{b} = {\int{{R_{b}(\lambda)}{\sum\limits_{i = 1}^{3}{ɛ_{i}{{E_{i}(\lambda)} \cdot {\sum\limits_{j = 1}^{3}{\sigma_{j}{S_{j}(\lambda)}{\mathbb{d}\lambda}}}}}}}}} \\{= {\sum\limits_{i = 1}^{3}{\sum\limits_{j = 1}^{3}{ɛ_{i}\sigma_{j}\left\{ {\int{{R_{b}(\lambda)}{E_{i}(\lambda)}{S_{j}(\lambda)}{\mathbb{d}\lambda}}} \right\}}}}}\end{matrix} & (6)\end{matrix}$Vectors ε_(v) and σ_(v) are defined as follows.ε_(v)=^(t)(ε₁, ε₂, ε₃)σ_(v)=^(t)(σ₁, σ₂, σ₃)  (7)When matrixes S_(r), S_(g) and S_(b) each having elements i and j aredefined by the following equations,Sr(i,j)=∫R _(r)(λ)E _(i)(λ)S _(j)(λ)dλSg(i,j)=∫R _(g)(λ)E _(i)(λ)S _(j)(λ)dλSb(i,j)=∫R _(b)(λ)E _(i)(λ)S _(j)(λ)dλ  (8)the values ρ_(r), ρ_(g) and ρ_(b) are expressed as follows.ρ_(r)=^(t)ε_(v) Srσ _(v)ρ_(g)=^(t)ε_(v) Sgσ _(v)ρ_(b)=^(t)ε_(v) Sbσ _(v)  (9)When a vector C is defined as follows,C= ^(t)(ρ_(r), ρ_(g), ρ_(b))  (10)the vector C is expressed in the following simplified form by using afunction F indicating three equations (9).C=F(ε_(v), σ_(v))  (11)From the pixel values of the first and second preliminary images withrespect to the target pixel, six nonlinear equations expressed asfollows are derived.C ₁ =F(ε _(v1), σ_(v))C ₂ =F(ε _(v2), σ_(v))  (12)

In the equations (12), the vector C₁ is a vector having elements of R, Gand B values ρ_(r1), ρ_(g1) and ρ_(b1) of the target pixel in the firstpreliminary image. The vector C₂ is a vector having elements of R, G andB values ρ_(r2), ρ_(g2) and ρ_(b2) of the target pixel in the secondpreliminary image. A vector ε_(v1) is a vector having weightingcoefficients ε₁₁, ε₁₂ and ε₁₃ as elements in the case of expressingillumination light to the subject by using basis functions. A vectorε_(v2) is a vector having weighting coefficients ε₂₁, ε₂₂ and ε₂₃related to illumination light as elements when the second preliminaryimage captured through the filter 116 is regarded as an image of thesubject irradiated with the illumination light, which is obtainedthrough the filter 116. That is, in the equations (12), an imageobtained through the filter 116 is regarded as an image of the subjectirradiated with different illumination light.

Since the spectral distribution of virtual illumination light in thesecond preliminary image is obtained by multiplying the spectraldistribution of actual illumination light by spectral transmittance ofthe filter 116, the vector ε_(v2) can be expressed by using the vectorε_(v1). In the six equations in the equations (12), only the sixelements of the vectors ε_(v1) ad σ_(v) are unknown.

Since the equation (11) has the term of ε_(pi)σ_(j) (p=1, 2), ε₁₁ isfixed to a predetermined value and the remaining five unknown elementsare obtained by the least square. That is, the vectors ε_(v1) and σ_(v)by which the following value (13) becomes the minimum are obtained.$\begin{matrix}{\mspace{20mu}{\sum\limits_{p = 1}^{2}\left\{ {C_{p} - {F\left( {ɛ_{vp},\sigma_{v}} \right)}} \right\}^{2}}} & (13)\end{matrix}$

By the method described above, the component data generating part 205obtains the vectors ε_(v1) and σ_(v) in each pixel from the first andsecond image data 231 and 232 while referring to the spectraltransmittance of the filter 116. The data of the spectral transmittanceof the filter 116 may be stored in the RAM 23 or a ROM in the componentdata generating part 205.

The vector σ_(v) of each pixel is stored as the object-color componentdata 235 into the RAM 23. The average value of the elements of thevector ε_(v1) is calculated with respect to all the pixels and is storedas the illuminant component data 236 into the RAM 23 together with thesense information. In the case where the illuminant component data 236is not demanded to have general versatility, the vector ε_(v1) of eachpixel may be stored as it is.

In the method as described above, for example, in the case where thefilter 116 is an ND filter having a uniform spectral transmittance in awaveband of visible light, the equations (12) cannot be solved.Therefore, the spectral transmittance of the filter 116 has to have theproperty such that the equation (12) can be solved. The spectraltransmittance has to be nonuniform at least in the waveband of each ofthe R, G and B colors.

FIG. 19 shows an example of the spectral transmittance of the filter 116satisfying such a condition. Reference numerals T11, T12 and T13 in FIG.19 denote graphs of spectral transmittance of on-chip filters of R, G anB colors formed on photosensitive devices of the CCD 112. Referencenumeral T2 denotes a graph of spectral transmittance of a movable filter116.

A filter formed by stacking thin films on a glass substrate is used asthe filter 116 and is designed so as to transmit light in a wavebandslightly deviated from the center of the waveband of each color of theon-chip filter in the CCD 112. Consequently, when the filter 116 isdisposed on the optical path for image capturing, the peak position ofthe spectral transmittance achieved by the filter 116 and the on-chipfilter is deviated from that of the spectral transmittance of theon-chip filter.

FIG. 20 is a graph showing spectral transmittances each obtained bymultiplying the spectral transmittance of the on-chip filter of each ofthe R, G and B colors shown in FIG. 19 by the spectral transmittance ofthe movable filter 116. Reference numerals T31, T32 and T33 in FIG. 20are graphs each showing the spectral transmittance in the case wherelight passes both the filter 116 and the on-chip filter of each of theR, G and B colors. As shown in FIG. 20, the peak position of thespectral transmittance (spectral sensitivity) of each of the R, G and Bon-chip filters after the light passes through the filter 116 isdeviated from that of the spectral transmittance of each of the R, G andB on-chip filters. Consequently, the object-color component data 235 andthe illuminant component data 236 can be calculated from the firstpreliminary image obtained without the filter 116 and the secondpreliminary image obtained with the filter 116 by the component datagenerating part 205.

The construction and operation of the digital camera 1 a according tothe second embodiment have been described above. The digital camera 1 aacquires the first preliminary image captured without the filter 116 andthe second preliminary image captured with the filter 116 by providingthe single movable filter 116, and obtains and stores the object-colorcomponent data 235 corresponding to image data of the subject obtainedby removing the influence of the illuminance environment from the aboveimages and the illuminant component data 236 indicative of the influenceof the illumination environment. By performing a reproducing operation(refer to FIG. 9) similar to that of the first embodiment, therefore, adesired target image under a desired illumination environment can bereproduced.

In a manner similar to the first embodiment, by providing standard light(D65 or the like) as the illuminant component data 236, an imagecaptured in an arbitrary illumination environment is reproduced by usingthe data of the standard light and accurate color reproduction of thetarget can be realized from the image. Further, by imparting the senseinformation 236 a to the illuminant component data 236, reproduction ofan image based on the sense can be enjoyed.

Although it has been described that the weighting coefficients σ₁, σ₂and σ₃ corresponding to each pixel are stored as the object-colorcomponent data 235 and the weighting coefficients ε₁, ε₂ and ε₃ commonto each pixel (peculiar to each pixel in the case where generalversatility is not required) are stored as the illuminant component data236, the data may include a basis function of the spectrum reflectivityof the subject and a basis function of a spectral distribution ofillumination light.

3. Third Embodiment

FIG. 21 shows the construction of a digital camera 1 b according to thethird embodiment. In the digital camera 1 a according to the secondembodiment, the single filter 116 can be disposed on the optical pathfor image capturing. The digital camera 1 b according to the thirdembodiment is different from the digital camera 1 a according to thesecond embodiment with respect to the point such that first and secondfilters 116 a and 116 b can be disposed on the optical path for imagecapturing by the operation of the motor 117. The internal constructionof FIG. 21 is similar to that of FIG. 16 except for the point that thefirst and second filters 116 a and 116 b can be disposed on the opticalpath for image capturing. FIG. 16 will be referred to with respect tothe construction other than the filters.

FIG. 22 is a block diagram showing the component data generating part205 realized by the CPU 21, the ROM 22, the RAM 23 and the like in thespecial image capturing mode for obtaining the object-color componentdata and the illuminant component data by the digital camera 1 btogether with the peripheral construction. As shown in FIG. 22, thethird embodiment is different from the second embodiment with respect tothe point that three image data 231, 232 a and 232 b are stored in theRAM 23 by the digital camera 1 b. FIG. 23 is a flow chart showing theflow of operations of the digital camera 1 b in the special imagecapturing mode.

In the operation of the digital camera 1 b, first, a color image iscaptured without a filter as a first preliminary image and is stored asthe first image data 231 into the RAM 23 (step ST51). Then the firstfilter 116 a is moved onto the optical path for image capturing, and asecond preliminary image is captured with the first filter 116 a and isstored as the second image data 232 a into the RAM 23 (step ST52).Further, the second filter 116 b is moved onto the optical path forimage capturing, and a third preliminary image is captured with thesecond filter 116 b and is stored as the third image data 232 b into theRAM 23 (step ST53). By the operations, the first to third image data231, 232 a and 232 b are stored into the RAM 23. The image capturingoperations of three times are performed sequentially and the subject ofimages is the same.

After that, the object-color component data 235 and the illuminantcomponent data 236 are obtained from the three image data by thecomponent data generating part 205 (step ST54). In a manner similar tothe first embodiment, after imparting the sense information to theilluminant component data 236 (step ST55), the object-color componentdata 235 and the illuminant component data 236 are stored into theexternal memory 123 (step ST56).

The principle of obtaining the object-color component data 235 and theilluminant component data 236 by the component data generating part 205in the digital camera 1 b will now be described.

The vectors C₁, C₂ and C₃ of the pixel values (R, G and B values) in thetarget pixel in the first to third preliminary images are expressed bythe following equations (14).C ₁ =F(ε _(v1), σ_(v))C ₂ =F(ε _(v2), σ_(v))C ₃ =F(ε _(v3), σ_(v))  (14)where, the vector ε_(v1) is a weighting coefficient vector of the basisfunction of illumination light in the case of using no filter, thevector ε_(v2) is a weighting coefficient vector of the basis function ofvirtual illumination light in the case of using the first filter, andthe vector ε_(v3) is a weighting coefficient vector of the basisfunction of virtual illumination light in the case of using the secondfilter. That is, the second preliminary image is regarded as an image ofthe subject irradiated with illumination light through the first filter116 a and the third preliminary image is regarded as an image of thesubject irradiated with illumination light through the second filter 116b. σ_(v) is a weighting coefficient vector of the basis function ofspectrum reflectivity in a position on the subject corresponding to thetarget pixel.

Since the vectors ε_(v2) and ε_(v3) are obtained from the vector ε_(v1)on the basis of the spectral transmittance of the first filter 116 a andthat of the second filter 116 b, six elements of the vectors ε_(v1) andσ_(v) are unknown in the nine equations in the equations (14).Consequently, the vectors ε_(v1) and σ_(v) are obtained by using theleast square so that the following becomes the minimum. $\begin{matrix}{\mspace{20mu}{\sum\limits_{p = 1}^{3}\left\{ {C_{p} - {F\left( {ɛ_{vp},\sigma_{v}} \right)}} \right\}^{2}}} & (15)\end{matrix}$In consideration of the fact that only six unknown elements exist, thevectors ε_(v1) and σ_(v) are obtained within a predetermined retrievalrange.

A specific example of the spectral transmittance of the first filter 116a and that of the second filter 116 b will now be described. In a mannersimilar to the second embodiment, a filter having uniform spectraltransmittance like an ND filter cannot be used as each of the first andsecond filters 116 a and 116 b. That is, a filter having spectraltransmittance which is not uniform at least in the waveband of each ofthe R, G and B colors has to be used.

In the third embodiment, therefore, a filter having spectraltransmittance shown by the reference numeral T2 in FIG. 19 is used asthe first filter 116 a, and a filter having spectral transmittanceobtained by shifting the spectral transmittance of the first filter 116a only by about 50 nm (to either the shorter wavelength side or longerwavelength side) is used as the second filter 116 b. By using thefilters, a larger amount of information of narrower half width, which isless overlapped and more independent can be obtained, so that theaccuracy of the object-color component data 235 and the illuminantcomponent data 236 can be improved.

Although the object-color component data 235 and the illuminantcomponent data 236 are obtained by using the three preliminary images inthe above description, arithmetic operations similar to those in thesecond embodiment may be executed by using only the second and thirdpreliminary images.

4. Fourth Embodiment

A digital camera according to a fourth embodiment has both theconstruction of the digital camera 1 according to the first embodimentand the construction of the digital camera 1 a according to the secondembodiment. The operations in the first and second embodiments areswitched and performed.

Since the construction of the digital camera is similar to that shown inFIGS. 15 and 16, it will be described with reference to FIGS. 15 and 16.The light emission control circuit 121 a shown in FIG. 16 is similar tothe light emission control circuit in the first embodiment and plays therole of maintaining the light emitting characteristic of the flash 121uniform.

FIG. 24 is a flow chart showing the flow of operations of the digitalcamera according to the fourth embodiment. In image capturing, first, animage capturing mode is determined (step ST61). Image capturing modesinclude: an ordinary image capturing mode for capturing an image by thedigital camera as an ordinary digital camera; a first image capturingmode for obtaining the object-color component data and the illuminantcomponent data with/without the flash 121 in a manner similar to thefirst embodiment; and a second image capturing mode for obtaining theobject-color component data and the illuminant component data dependingon whether the filter 116 is disposed on the optical path for imagecapturing or not in a manner similar to the second embodiment. The imagecapturing mode may be determined manually or automatically as will bedescribed hereinlater.

When the determination of the first or second image capturing mode isdone manually, for example, in the case where the operator judges thatthe distance to the subject is long (that is, flash light does notreach), the operator selects the second image capturing mode. When theoperator judges that the subject is not irradiated with sufficientlight, the operator selects the first image capturing mode. There is noclear criterion of judgement. The operator properly determines the modein consideration of the image capturing environment.

After the image capturing mode is determined, the digital cameracaptures an image in the selected image capturing mode. Specifically, inthe case of the ordinary image capturing mode, an image is captured inan ordinary manner, and image data in which pixels values are expressedby R, G and B values is stored into the external memory 123 (steps ST62and ST63). In the case of the first image capturing mode, the operationsshown in FIG. 5 are performed and the object-color component data andthe illuminant component data are stored into the external memory 123(steps ST64 and ST65). In the case of the second image capturing mode,the operations shown in FIG. 18 are performed and the object-colorcomponent data and the illuminant component data are stored into theexternal memory 123 (steps ST64 and ST66).

As described above, in the digital camera according to the fourthembodiment, the image capturing mode can be selected according to theimage capturing environment at the time of obtaining the object-colorcomponent data and the illuminant component data. As a result, theobject-color component data and the illuminant component data can beproperly obtained.

A method of automatically determining either the first image capturingmode or the second image capturing mode will now be described. It isassumed that whether the ordinary image capturing mode is selected ornot is set manually.

FIG. 25 is a block diagram showing the construction of automaticallydetermining the first or second image capturing mode. The imagecapturing mode is determined by a mode determining part 206. A distanceto the subject is inputted from the range sensor 114 to the modedetermining part 206 and brightness of the subject is inputted from aphotometer 118. As shown in FIG. 15, the range sensor 114 is disposed inthe upper part of the front face of the lens unit 11, and the photometer118 is also disposed in the window in which the range sensor 114 isdisposed. The function of the photometer 118 may be realized by the CCD112.

The mode determining part 206 shown in FIG. 25 is a part of thefunctions realized by the CPU 21, the ROM 22, the RAM 23 and the like.The operation of the mode determining part 206 is realized in such amanner that signals from the range sensor 114 and the photometer 118 aresupplied to the CPU 21 and the CPU 21 performs an arithmetic operationin accordance with the program 221 in the ROM 22.

When the distance to the subject is equal to or shorter than apredetermined distance, the mode determining part 206 selects the firstimage capturing mode. When the distance to the subject is longer thanthe predetermined distance, the mode determining part 206 selects thesecond image capturing mode. When the distance to the subject exceedsthe predetermined distance and the brightness of the subject is lowerthan predetermined brightness, the operator is notified through thedisplay 125 that the image capturing cannot be performed properly. Inthis case, the operator forcedly determines an image capturing mode.

By the automatic selection between the first and second image capturingmodes by the mode determining part 206 as described above, theobject-color component data and the illuminant component data can beproperly obtained according to the image capturing environment.

5. Fifth Embodiment

In the digital camera 1 according to the first embodiment, the lightemitting characteristic of the flash 121 is maintained uniform by thelight emission control circuit 121 a to thereby make the spectraldistribution of flash light uniform. Consequently, the object-colorcomponent data can be stably obtained. In the fifth embodiment as amodification of the first embodiment, a method of obtaining a relativespectral distribution of flash light by monitoring the light emittingstate of the flash light and obtaining the object-color component dataand the illuminant component data on the basis of the obtaineddistribution will be described. Since the main construction of thedigital camera is similar to that in FIGS. 1 and 2, the drawings will bereferred to.

FIG. 26 shows the construction of obtaining the relative spectraldistribution of flash light (that is, the flash spectral data 234 inFIG. 4) in the digital camera according to the fifth embodiment andmainly illustrates the part different from the construction of FIG. 4. Aflash spectral data group 234 a in the RAM 23 shown in FIG. 26 is agroup of data indicative of the relative spectral distributions of flashlight under a plurality of representative light emitting conditions.Representative light emitting conditions of total four kinds are suchthat, for example, a charging voltage is 330V or 250V and light emittingtime is 1 or 1/32 (it is assumed that the longest light emitting time is1). FIG. 27 shows a part of an example of the flash spectral data group234 a. The solid graph having solid rhombuses indicates a relativespectral distribution of flash light in the case where the chargingvoltage is 330V and the light emitting time is 1. The solid line havingblank triangles indicates a relative spectral distribution of flashlight in the case where the charging voltage is 330V and the lightemitting time is 1/32.

In the digital camera according to the fifth embodiment, the lightemission control circuit 121 a does not have the function of maintainingthe light emitting characteristic of the flash 121 uniform as in thefirst embodiment. The light emission control circuit 121 a controls thelight emission of the flash 121 so as to achieve a proper light emittingamount on the basis of information from the range sensor 114 and thephotometer 1118. In this case, the light emitting state of the flash 121is monitored.

An interpolating part 207 is a function realized by the CPU 21, the ROM22, the RAM 23 and the like and interpolates the data in the flashspectral data group 234 a on the basis of the light emitting statemonitored by the light emission control circuit 121 a, therebygenerating the flash spectral data 234.

FIG. 28 is a flow chart showing the flow of operations of the digitalcamera at the time of image capturing. Operations subsequent to stepST73 are similar to those subsequent to step ST13 in FIG. 5.

First, an image capturing operation is performed with the flash 121 anda first image is stored as the first image data 231 into the RAM 23(step ST71). At this time, the light emission control circuit 121 amonitors the light emitting state of the flash 121, and the chargingvoltage supplied to the power source of the flash 121 at the time oflight emission and the light emission time of the flash 121 are sent tothe interpolating part 207. Subsequently, a second image is capturedwithout the flash 121 and is stored as the second image data 232 intothe RAM 23 (step ST72).

After the two image data is obtained, the data in the flash spectraldata group 234 a is interpolated by the interpolating part 207 on thebasis of the light emitting state of the flash 121, and the relativespectral distribution of flash light is obtained as the flash spectraldata 234 and stored into the RAM 23 (step ST73).

For example, when the charging voltage at the time of light emission ofthe flash 121 is 330V and the light emission time is ½, the graph of thecharging voltage of 330 V and the light emission time of 1 and the graphof the charging voltage of 330V and the light emission time of 1/32 areinterpolated by using the light emission time as a reference and thebroken-line graph having blank squares is obtained as a relativespectral distribution of flash light in the case where the chargingvoltage is 330V and the light emission time is ½. Interpolation such aslinear interpolation, linear interpolation after weighting, ornon-linear interpolation is carried out.

The flash spectral data 234 may be calculated either at the stage ofstep ST71 or the stage of step ST14 (FIG. 5).

After that, in a manner similar to the first embodiment, theobject-color component data 235 and the illuminant component data 236are obtained and stored into the external memory 123 (steps ST13 toST17).

The digital camera according to the fifth embodiment has been describedabove. In the digital camera, the light emitting state of the flash 121is monitored, and the flash spectral data group 234 a is interpolated inaccordance with the light emitting state (that is, actual light emittingconditions), thereby obtaining the flash spectral data 234.Consequently, while properly using the flash 121 in accordance with theimage capturing environment, the object-color component data 235 and theilluminant component data 236 can be properly acquired.

At the time of obtaining the flash spectral data 234, the data in theflash spectral data group 234 a is interpolated. The quantity of theflash spectral data group 234 a to be prepared can be therefore reduced.

6. Sixth Embodiment

In the fifth embodiment, by interpolating the relative spectraldistributions of flash light under the representative light emittingconditions, the relative spectral distribution of actual flash light isobtained. In the sixth embodiment, a method of preparing a database ofthe relative spectral distributions of flash light under more detailedlight emitting conditions and determining the relative spectraldistribution of flash light from the light emission state of the flash121 will be described.

FIG. 29 shows the construction for determining the relative spectraldistribution of flash light and corresponds to FIG. 26 in the fifthembodiment. The other construction is similar to that of the fifthembodiment, and the reference numerals shown in FIGS. 1 to 4 will beproperly referred to.

A flash spectral database 234 b in FIG. 29 is a database of relativespectral distributions of flash light under various light emissionconditions. Table 1 shows an example of the flash spectral database 234b.

TABLE 1 light charging emission wavelength[nm] voltage[v] time 400 420440 460 680 700 1 330 1 0.1 0.2 0.4 0.5 0.7 0.65 2 330 ½ 0.1 0.22 0.430.55 0.68 0.62 3 330 ¼ 0.1 0.23 0.44 0.57 0.65 0.61 4 330 ⅛ 0.13 0.260.48 0.59 0.63 0.6 21 290 1 0.1 0.3 0.45 0.57 0.7 0.65 22 290 ½ 0.1 0.310.48 0.59 0.71 0.68 23 290 ¼ 0.1 0.33 0.51 0.6 0.76 0.73 24 290 ⅛ 0.10.35 0.55 0.63 0.78 0.7 41 250 1 0.1 0.33 0.58 0.54 0.68 0.62 42 250 ½0.1 0.35 0.6 0.53 0.69 0.6

Table 1 shows the flash spectral database 234 b including relativespectral distributions of flash light in various combinations of thecharging voltage to the power source of the flash 121 and the lightemission time.

A spectral distribution determining part 208 is a function realized bythe CPU 21, the ROM 22, the RAM 23 and the like and determines the mostproper relative spectral distribution of flash light from the flashspectral database 234 b on the basis of the light emitting state of theflash 121 received from the light emission control circuit 121 a.

For example, when the charging voltage is 325V and the light emissiontime is ⅕, the relative spectral distribution of flash light with thecharging voltage of 330V and the light emission time of ¼ as the closestlight emission conditions is determined as an actual relative spectraldistribution.

The operation of the digital camera in the sixth embodiment is almostsimilar to that of the digital camera in the fifth embodiment. In thesixth embodiment, step ST73 in FIG. 28 is replaced by step ST74 in FIG.30. That is, first, while monitoring the light emitting state of theflash 121, the first image data 231 is obtained with the flash 121 (stepST71) and then the second image data 232 is obtained without the flash121 (step ST72).

After obtaining two images, the spectral distribution determining part208 extracts the optimum relative spectral distribution from the flashspectral database 234 b on the basis of the light emitting statemonitored by the light emission control circuit 121 a (step ST74). Insuch a manner, data corresponding to the flash spectral data 234 in thefirst embodiment is supplied to the object-color component datagenerating part 202.

Subsequent to step ST74, operations similar to those subsequent to stepST13 in the first embodiment are performed, so that the object-colorcomponent data 235 and the illuminant component data 236 are obtainedand stored into the external memory 123 (steps ST13 to ST17).

The digital camera according to the sixth embodiment has been describedabove. In the digital camera, the relative spectral distribution offlash light used for an arithmetic operation is determined by referringto the flash spectral database 234 b. Consequently, even in the casewhere the operation of the flash 121 is not controlled fixedly, therelative spectral distribution of flash light can be promptlydetermined.

7. Seventh Embodiment

Although image data is processed in the digital camera in the foregoingembodiments, obviously, it can be performed by a computer. FIG. 31 showsthe construction of an image data obtaining system 3 in such a case.

The image data obtaining system 3 comprises: a digital camera 31 forstoring image data acquired by the CCD as it is into an external memory;and a computer 32 for processing the image data stored in the externalmemory to thereby obtain the object-color component data and theilluminant component data. Such a construction can be used for any ofthe operations in the first to sixth embodiments.

For example, in the case of using the image data obtaining system 3shown in FIG. 31 for the processes of the first embodiment, the digitalcamera 31 having the construction shown in FIG. 3 is used, the firstimage data obtained by performing image capturing with a flash, thesecond image data obtained by performing image capturing without aflash, and flash spectral data related to flash light is stored into theexternal memory 123, and those data is transferred via the externalmemory 123 to the computer 32.

The CPU, ROM, RAM and the like in the computer 32 function as thedifferential image generating part 201, the object-color component datagenerating part 202 and the illuminant component data generating part203 shown in FIG. 4 to calculate the object-color component datacorresponding to image data obtained by removing the influence of theillumination environment from the first image data, the second imagedata and the flash spectral data and the illuminant component datacorresponding to the components of the illumination environment.

In this case, it is unnecessary for the digital camera 31 to have thefunction of obtaining the object-color component data and the illuminantcomponent data.

In the case of using the image data obtaining system 3 for the processesin the second embodiment, the digital camera 31 having the constructionshown in FIG. 16 is used. The first image data obtained by performingthe image capturing without the filter, the second image data obtainedby performing the image capturing with the filter, and the data of thespectral transmittance of the filter 116 is stored in the externalmemory 123 and those data is transferred to the computer 32.

The CPU, ROM, RAM and the like in the computer 32 function as thecomponent data generating part 205 shown in FIG. 17, and calculate theobject-color component data and the illuminant component data from thefirst image data, the second image data and the spectral transmittanceof the filter 116. By prestoring the spectral transmittance of thefilter 116 in the computer 32, it can be arranged to transfer only thefirst image data and the second image data from the digital camera 31.

In this case as well, the function of obtaining the object-colorcomponent data and the illuminant component data is unnecessary in thedigital camera 31.

In order to make the computer 32 function as the differential imagegenerating part 201, the object color component data generating part 202and the illuminating component data generating part 203 shown in FIG. 4or the component data generating part 205 shown in FIG. 17, a program ispreliminarily installed into the computer 32 via a recording medium 9such as a magnetic disk, an optical disk, a magnetooptic disk, or thelike. Consequently, a general computer 32 can be used as the computerfor performing the image process.

The operation for reproducing an image shown in FIG. 9 can be realizedby the computer 32 by installing a program for a reproducing operationinto the computer 32.

As described above, the digital camera according to any one of the firstto sixth embodiments can be used in the image data obtaining system 3comprised of the digital camera 31 and the computer 32. In this case, anamount of processes to be performed by the digital camera 31 can bereduced.

8. Modification

Although the embodiments have been described above, the embodiments canbe variously modified.

For example, any illuminant component data in the foregoing embodimentscan be used as long as data indicates an influence of the illuminationenvironment on an image. The illuminant component data is not strictlyrequired to indicate the influence of the illumination environment butmay indicate the influence of the illumination environment to someextent. Any data indicative of the components obtained by removing theinfluence of the illumination environment from an image can be used asthe object-color component data. The data does not always have toindicate the components obtained by strictly removing the influence ofthe illumination environment from an image.

Although it has been described in the foregoing embodiments that theobject-color component data and the illuminant component data is storedas a plurality of weighting coefficients, other formats of storing thedata may be also employed. For example, the object-color component datamay be stored as a characteristic curve of spectrum reflectivity and theilluminant component data may be stored as a characteristic curve of aspectral distribution.

It is not always necessary to use the digital camera 1 or the image dataobtaining system 3 to obtain the illuminant component data. For example,illuminant component data intended for virtual illumination may begenerated separately. Further, the illuminant component data used at thetime of reproducing an image may be generated by combining a pluralityof illuminant component data.

Although it has been described that a movable filter is provided in thedigital camera in each of the second to fourth embodiments, the filtermay be disposed on the optical path for image capturing by the operator.For example, as shown in FIG. 32, it is also possible to prepare anattachable filter 119 having spectral transmittance similar to that ofthe filter 116 in front of the lens system 111 and to obtain a pluralityof preliminary images by attaching and detaching the filter 119 by theoperator. In this case, in order to make the subject the same in theplurality of preliminary images, the digital camera is fixed. When thefilter is attached/detached manually in the fourth embodiment, by theselection of the second image capturing mode, the digital cameranotifies the operator that the filter 116 has to be attached by usingthe display 125 or an alarm and it is controlled so that the shutter isnot released unless the filter is attached.

Although the single filter 116 is used in the second embodiment and thetwo filters 116 a and 116 b are used in the third embodiment, three ormore filters may be also used. That is, if at least one filter can bedisposed on the optical path for image capturing, by obtaining aplurality of image data while changing the disposing state of the filteron the optical path for image capturing, the object-color component dataand the illuminant component data can be obtained.

The method of obtaining the object-color component data and theillumination component data in the second to fourth embodiments are notlimited to the foregoing embodiments but other methods may be employed.

Although the light emission characteristics of the flash 121 are keptuniform by monitoring the charging voltage and the light emission timein the first embodiment, the light emission characteristics of the flash121 may be kept uniform by other methods. For example, by emitting theflash 121 in pulses, the light emission characteristics of the flash maybe kept uniform.

Although the light emission control circuit 121 a monitors the chargingvoltage and the light emission time in the first, fourth and sixthembodiments, if the spectral characteristics of the flash 121 can bekept uniform or the spectral characteristics can be determined, otherlight emitting states (light emitting conditions) can be monitored.

Although the flash 121 functions as means for changing the illuminationenvironment of the subject in the first, fourth and six embodiments, themethod of changing the illumination environment is not limited to themethod of using the flash 121.

In the embodiments, it has been described that the CCD 112 has threeband inputs of R, G and B. However, the number of input bands may befour or more. That is, when an image obtained by the CCD 112 (which maybe what is called a 3 CCD) substantially corresponds to a color image,the object-color component data and the illumination component data canbe obtained by the above-described method.

While the invention has been shown and described in detail, theforegoing description is in all aspects illustrative and notrestrictive. It is therefore understood that numerous modifications andvariations can be devised without departing from the scope of theinvention.

1. An image capturing apparatus comprising: an illuminator for changingan illumination environment around a subject, wherein the illuminatorhas a light emitting part emitting light to said subject; an imagesensor for obtaining an image of said subject; a first memory formemorizing first image data obtained by said image sensor withilluminating by said illuminator; a second memory for memorizing secondimage data obtained by said image sensor without illuminating by saidilluminator; a difference data calculator for calculating differencedata between said first image data memorized in said first memory andsaid second image data memorized in said second; a spectral reflectivitycalculator for calculating spectral reflectivity of said subject basedon the difference data and relative spectral distribution of light fromsaid light emitting part to obtain subject data; and an illuminationdata calculator for calculating illumination data on the basis of thesecond image data and said subject data, wherein said illumination dataindicates influence of an illumination environment on the second imagedata.
 2. The apparatus of claim 1, further comprising a controller forcontrolling lighting of said light emitting part to keep spectraldistribution of light from said light emitting part constant.
 3. Theapparatus of claim 1, further comprising: a third memory for memorizinga plurality of illumination data each of which is different from oneanother; a selector for selecting any one of said plurality ofillumination data; and a reproducer for reproducing a subject image onthe basis of illumination data selected by said selector and saidsubject data.
 4. The apparatus of claim 3, wherein each of saidplurality of illumination data is given sense information whichrepresents a sense given to an observer observing an image influenced bya corresponding illumination environment, and said selector displays aplurality of pieces of sense information.
 5. A method of shooting asubject by an image capturing apparatus which has a light emitting partemitting light to said subject, said method comprising the steps of:obtaining first image data by shooting said subject with an image sensorunder irradiation of light from said light emitting part; obtainingsecond image data by shooting said subject with said image sensorwithout irradiation of light from said light emitting part; calculatingdifference data between said first image data and said second imagedata; and calculating spectral reflectivity of said subject based on thedifference data and relative spectral distribution of light from saidlight emitting part to obtain subject; and calculating illumination dataon the basis of the second image data and said subject data, whereinsaid illumination data indicating influence of said illuminationenvironment on the second image data.